Search for: All records

Abstract In recent years, large language models (LLMs) and vision language models (VLMs) have excelled at tasks requiring human-like reasoning, inspiring researchers in engineering design to use language models (LMs) as surrogate evaluators of design concepts. But do these models actually evaluate designs like humans? While recent work has shown that LM evaluations sometimes fall within human variance on Likert-scale grading tasks, those tasks often obscure the reasoning and biases behind the scores. To address this limitation, we compare LM word embeddings (trained to capture semantic similarity) with human-rated similarity embeddings derived from triplet comparisons (“is A closer to B than C?”) on a dataset of design sketches and descriptions. We assess alignment via local tripletwise similarity and embedding distances, allowing for deeper insights than raw Likert-scale scores provide. We also explore whether describing the designs to LMs through text or images improves alignment with human judgments. Our findings suggest that text alone may not fully capture the nuances humans key into, yet text-based embeddings outperform their multimodal counterparts on satisfying local triplets. On the basis of these insights, we offer recommendations for effectively integrating LMs into design evaluation tasks. 

ABSTRACT: The Consensual Assessment Technique (CAT) is one of the most effective and commonly used design evaluation methods. However, it fails to capture implicit cognitive processes and has mainly been studied in a homogenous design modality. To bridge this gap, the present study investigates the impact of design ideas represented in different modalities (i.e., text-only, sketch-only, text + sketch) on design evaluations for creativity, novelty, and usefulness, and examine human gaze patterns during the evaluation process. Our findings showed that novice raters exhibit higher interrater reliability and greater convergence in visual attention when rating ideas containing sketches compared to text-only design modality, highlighting the value of visual elements in design evaluations. 

Abstract Applications for additive manufacturing (AM) continue to increase as more industries adopt the technology within their product development processes. There is a growing demand for designers to acquire and hone their design for AM (DfAM) intuition and generate innovative solutions with AM. Resources that promote DfAM intuition, however, historically default to physical or digitally non-immersive modalities. Immersive virtual reality (VR) naturally supports 3D spatial perception and reasoning, suggesting its intuitive role in evaluating geometrically complex designs and fostering DfAM intuition. However, the effects of immersion on DfAM evaluations are not well-established in the literature. This study contributes to this gap in the literature by examining DfAM evaluations for a variety of designs across modalities using varying degrees of immersion. Specifically, it observes the effects on the outcomes of the DfAM evaluation, the effort required of evaluators, and their engagement with the designs. Findings indicate that the outcomes from DfAM evaluations in immersive and non-immersive modalities are similar without statistically observable differences in the cognitive load experienced during the evaluations. Active engagement with the designs, however, is observed to be significantly different between immersive and non-immersive modalities. By contrast, passive engagement remains similar across the modalities. These findings have interesting implications on how organizations train designers in DfAM, as well as on the role of immersive modalities in design processes. Organizations can provide DfAM resources across different levels of immersion, enabling designers to customize how they acquire DfAM intuition and solve complex engineering problems. 

Abstract Well-studied techniques that enhance diversity in early design concept generation require effective metrics for evaluating human-perceived similarity between ideas. Recent work suggests collecting triplet comparisons between designs directly from human raters and using those triplets to form an embedding where similarity is expressed as a Euclidean distance. While effective at modeling human-perceived similarity judgments, these methods are expensive and require a large number of triplets to be hand-labeled. However, what if there were a way to use AI to replicate the human similarity judgments captured in triplet embedding methods? In this paper, we explore the potential for pretrained Large Language Models (LLMs) to be used in this context. Using a dataset of crowdsourced text descriptions written about engineering design sketches, we generate LLM embeddings and compare them to an embedding created from human-provided triplets of those same sketches. From these embeddings, we can use Euclidean distances to describe areas where human perception and LLM perception disagree regarding design similarity. We then implement this same procedure but with descriptions written from a template that attempts to isolate a particular modality of a design (i.e., functions, behaviors, structures). By comparing the templated description embeddings to both the triplet-generated and pre-template LLM embeddings, we identify ways of describing designs such that LLM and human similarity perception better agree. We use these results to better understand how humans and LLMs interpret similarity in engineering designs. 

Abstract Solving problems with additive manufacturing (AM) often means fabricating geometrically complex designs, layer-by-layer, along one or multiple directions. Designers navigate this 3D spatial complexity to determine the best design and manufacturing solutions to produce functional parts, manufacturable by AM. However, to assess the manufacturability of their solutions, designers need modalities that naturally visualize AM processes and the designs enabled by them. Creating physical parts offers such visualization but becomes expensive and time-consuming over multiple design iterations. While non-immersive simulations can alleviate this cost of physical visualization, adding digital immersion further improves outcomes from the visualization experience. This research, therefore, studies how differences in immersion between computer-aided (CAx) and virtual reality (VR) environments affect: 1. determining the best solution for additively manufacturing a design and 2. the cognitive load experienced from completing the DfAM problem-solving experience. For the study, designers created a 3D manifold model and simulated manufacturing it in either CAx or VR. Analysis of the filtered data from the study shows that slicing and printing their designs in VR yields a significant change in the manufacturability outcomes of their design compared to CAx. No observable differences were found in the cognitive load experienced between the two modalities. This means that the experiences in VR may influence improvements to manufacturability outcomes without changes to the mental exertion experienced by the designers. This presents key implications for how designers are equipped to solve design problems with AM. 

Parents boost STEM skills by scaffolding children’s attention and discovery during play, but many need support to do so. Using Human Centered Design (HCD) methods, we created activity kits fostering parents’ (a) involvement in and (b) valuing of parent-child play to promote preschoolers’ STEM skills. Study 1 documents how HCD methods informed the design of guided activity kits. In initial home visits, we videorecorded 6 parent-child dyads playing with basic building materials. Play revealed minimal parental STEM scaffolding and talk. Collaborating with 18 families and drawing on prior research, parent interviews, videotaped play sessions, and advisory-board members’ expertise, the interdisciplinary research team designed and refined activity kit prototypes. Study 2 was a randomized field test comparing use and evaluation of final guided kits (n=50) versus basic kits (n=25) which contained identical building materials and challenges but omitted scaffolding guides. Both groups received a kit by mail every other week for 10 weeks. Relative to parents given basic kits, parents given guided kits (a) reported significantly more sustained use of the kits across the 10 weeks, (b) felt more self-efficacy in fostering their child’s STEM learning, and (c) judged that their child had achieved greater STEM-skill learning from program use. 

Traditional lectures have difficulties instilling pragmatic skills in construction engineering students due to the inability to illuminate the complexities within the human-robot collaborative construction environment. While on-site can acclimatize construction students to reality and construct knowledge that can solve safety challenges, it is challenging to organize on-site training trips owing to the dangerous nature of construction workplaces. This research aimed to explore virtual reality (VR) as a tool to enhance students’ perception and knowledge of construction robotic safety. For this purpose, the study developed a virtual training platform for providing construction engineering students with safety knowledge on interacting with simulated robots within the virtual environment of construction sites. A self-assessment approach was leveraged among 20 recruited students to demonstrate the efficacy of students’ engagement and learning outcomes from the proposed learning approach over the traditional learning approach. Results indicated a statistical difference in students’ learning outcomes and engagement levels between the developed approach and the traditional approach. Findings demonstrated the implications of VR as an experiential tool to enhance the students’ learning of robotic safety in construction.